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This report summarizes the methodology, measures, and findings of research on the 
influence on student achievement outcomes of K-12 online and blended face-to-face 
and online learning programs that offer differentiated learning options. The report also 
describes the characteristics of the learning programs. Most of the examined programs 
used blended learning strategies, and all the examined programs offered some means 
to differentiate their content, difficulty level, or pacing. Some 45 percent of the blended 
learning programs studied (5 of 11) offered differentiation in the face-to-face component 
of instruction. In some of the most rigorous studies, statistically significant positive 
effects were found for four blended learning programs: Cognitive Tutor Algebra I, 
LeapTrack, READ 180, and Time To Know. 


Why this study? 


As more rigorous college and career readiness standards are established, new instructional models or 
practices — including online and blended learning programs — may be needed to help high school stu- 
dents learn and graduate with the knowledge and skills targeted by these standards (Knowledge Works, 
2013). Members of the Regional Educational Laboratory Central College and Career Readiness Research 
Alliance asked Regional Educational Laboratory Central for a summary of research on K-12 online and 
blended learning programs that offer options for differentiated instruction, referred to in this report as 
differentiated learning options. 

Differentiated learning options provide a variety of ways for students to engage with content and acquire 
knowledge and can be tailored to student interests and academic skills (Tomlinson, 2001). Proponents of 
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differentiated learning options argue that students differ in learning styles, knowledge and skills, and learn- 
ing pace and that students learn best when instruction is aligned to their interests and needs (Tomlinson, 
2000 ). 

Online and blended learning programs can offer differentiated learning options. The programs can be fully 
online, meaning that content and instruction are delivered through the Internet, or blended, meaning that 
programs use both online and face-to-face instruction (Means, Toyama, Murphy, & Baki, 2013; Watson, 
Gemin, & Coffey, 2010). Some online and blended learning programs include proprietary packaged cur- 
ricula or computer software that teachers use to offer differentiated learning options. For example, blended 
learning programs frequently use adaptive programs for the online component to assess students’ knowl- 
edge and skills; adaptive programs can also monitor a student’s progress and can tailor the content, pacing, 
or difficulty of the online instruction to a student’s current needs. Online and blended learning programs 
can also allow students to progress through the online content at a flexible pace, giving students sufficient 
time to master the content. But not all online and blended learning programs offer differentiated learning 
options. Some online programs are organized like traditional face-to-face courses, where the sequence and 
pacing of topics are identical for all students. 

Meta-analyses and literature reviews have generally found online and blended learning programs to be as 
effective as or more effective than traditional face-to-face instruction (Means, Toyama, Murphy, Bakia, & 
Jones, 2009; Sitzmann, Kraiger, Stewart, & Wisher, 2006; Wisher & Olson, 2003). However, the majority 
of research has been conducted with populations other than K-12 students, such as students at online uni- 
versities and adult technical education programs. For example, in one meta-analysis of online and blended 
learning programs, only 5 of 176 studies that met the authors’ eligibility criteria focused on K-12 schooling 
(Means et al., 2009). And no previous meta-analysis or literature review has focused on online and blended 
learning programs that offer differentiated learning options. 

School and district leaders may be able to use the information in this report to inform their selection or 
design of online or blended learning programs to meet the needs of their students. And researchers may 
be able to use the information in this report to inform the design of future studies of online and blended 
learning programs that offer differentiated learning options. 

What the study examined 


This study addressed two research questions: 

• What are the characteristics of online and blended learning programs that offer differentiated 
learning options to K-12 students in the United States and that have been the subject of research? 

• What are the methodologies, measures, findings, and strength of evidence of studies that have 
examined these programs using randomized controlled trials, quasi-experimental, or correlational 
research designs? 

To answer these questions, the study team searched research journal databases, research organizations’ 
websites, and the websites of online and blended learning program vendors to identify relevant studies. 
Studies were included only if they focused on online or blended learning programs that offered differen- 
tiated learning options (see box 1 for definitions of key terms used in the report). Programs were included 
regardless of whether they were used to provide a full curriculum (including content for a semester or year- 
long course) or to supplement an existing course curriculum. 

Because this study focused on programs that offer differentiated learning options, studies of standalone pro- 
grams that did not involve a classroom teacher were excluded. Studies were included only if the programs 
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Box 1. Key terms 


Blended learning program. A program in which instruction occurs through a combination of face-to-face instruc- 
tion and online learning. The online learning component frequently uses proprietary, online, adaptive software. 

Correlational research studies. Studies that assess the relationship between the implementation of online and 
blended learning programs and student achievement outcomes for a single group of students. 

Differentiated learning options. Educational opportunities in which course content or instruction is tailored 
to meet individual student learning needs. For the current study, differentiated learning options in online and 
blended learning programs included face-to-face, individual, or small-group instruction; teacher or computerized 
selection of student-individualized educational content or difficulty level; and flexible time for students to master 
course content. 

Full curriculum. An online or blended learning program that provides a full course curriculum. 

Online learning program. A program in which instruction occurs primarily online, with no face-to-face interaction 
between the teacher and students. Teachers monitor student progress throughout the program, provide feed- 
back on assignments, and communicate with students through telephone or email. 

Quasi-experimental design studies. Studies in which students or classrooms that used an online or blended 
learning program are compared with students or classrooms that did not use such a program and in which the 
students or classrooms are not randomly assigned to the treatment or comparison group. 

Randomized controlled trial studies. Studies in which students or classrooms are randomly assigned to partici- 
pate in an online or blended program and compared with students or classrooms that are randomly assigned to 
a comparison group. 

Standalone program. A program in which a course instructor has no active role in the instruction provided. This 
includes supplemental and intervention programs that are intended to develop targeted knowledge or skills but 
that are not aligned to the in-class curriculum. Homework and pull-out programs are also classified as stand- 
alone programs. These programs were excluded from the current study. 

Supplemental curriculum An online or blended learning program that supplements an existing curriculum. Pro- 
grams that were not integrated into the regular course curriculum (for example, those used as standalone inter- 
ventions) were excluded from the study. 

What Works Clearinghouse (WWC) standards. The WWC has described standards for determining whether a 
study is well conducted and whether causal inferences can be made from it (What Works Clearinghouse, 2014). 
For a randomized controlled trial study to meet WWC standards without reservations, student attrition in the 
treatment and comparison groups must be low. For a randomized controlled trial study with high attrition or a 
quasi-experimental design study to meet WWC standards with reservations, the treatment and comparison 
groups must be equivalent on the outcome measure prior to the treatment. 


they examined directly involved a teacher in the delivery and monitoring of instruction and the study 
report clearly stated that the online or blended learning program being examined was used to support 
differentiated instruction. Online programs that were completely software-driven and that did not have a 
teacher facilitator were excluded. See appendix A for details on the screening and review process. 

The study team reviewed 162 unique studies. Seventeen of the studies met the current study’s eligibility cri- 
teria. Of these, 15 used a randomized controlled trial or quasi-experimental design and 2 were correlational 
studies. Some online and blended learning programs were the subject of multiple studies, and some studies 
examined multiple programs. The 17 studies examined 14 different online or blended learning programs. 
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(See table B1 in appendix B for details about each of the programs, including program type, form of online 
or in-class differentiation, course content, and course grade level, and see table B2 for details about each 
of the studies, including research methods used, independent and dependent measures, study findings, and 
study design rating.) 

The 15 studies that used a randomized controlled trial or quasi-experimental design were rated using an 
approach modeled after the U.S. Department of Education’s What Works Clearinghouse (WWC) evidence 
standards and procedures for identifying studies that support causal inferences (What Works Clearing- 
house, 2014). Ratings were determined by WWC-certified members of the study team or taken from prior 
WWC intervention reports. 1 Each study was assigned one of three ratings: meets standards without reser- 
vations, meets standards with reservations, or does not meet standards. If a study already had a rating from 
a WWC intervention report, that rating was used for this study. 2 Each study that was rated by the study 
team as meeting WWC standards was also rated by a second WWC-certified member of the team to ensure 
inter-rater reliability. 


What the study found 


This section describes the characteristics of the 14 online and blended learning programs that were the 
subject of research studies that met the current study’s eligibility criteria, as well as the characteristics of the 
17 studies that examined these programs. 

Characteristics of online and blended learning programs 

Of the 14 online and blended learning programs with studies that met the current study’s eligibility criteria, 
11 used a blended learning strategy, either to supplement an existing in-class curriculum or to provide a 
full blended learning curriculum (figure 1). The other three programs used a full curriculum online model. 
Nine of the programs were developed for elementary grades, and five were developed for middle school or 
high school courses (for example, Algebra I). Five programs addressed only English language arts, three 
addressed only math, and six addressed multiple content areas. 


Figure 1. Of the 14 online and blended learning programs with studies that met the current study’s 
eligibility criteria, 11 used a blended learning strategy, and 3 used a full curriculum online model 


Number of programs 



Supplemental curriculum, Full curriculum, Full curriculum, 

blended blended online 


Source: Authors' analysis in November 2015 following the procedures described in appendix A. 
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The online and blended learning programs were classified according to the involvement of the teacher 
and how the online and face-to-face components were integrated. In the 11 blended learning programs 
all communications between teachers and students were face to face (there was no online interaction). 
However, teachers received reports of student progress in the online portion of the course, which the teach- 
ers could use to inform the in-class instruction. In the three online programs, communication occurred 
only through telephone calls, email, or teacher notes on graded assignments (there was no real-time online 
or face-to-face student-teacher interaction). The five supplemental curriculum blended learning programs 
(for example, Waterford Early Reading) were frequently used to support the development of students’ early 
literacy skills within the face-to-face classroom. The six full curriculum blended learning programs pro- 
vided online and in-class instructional materials. All three full curriculum online learning programs were 
completely online programs offered by virtual schools, with a teacher facilitating the course and providing 
feedback to students. 

Online and blended learning programs varied in how they offered differentiated learning options. Each 
of the 14 online and blended learning programs was characterized by the types of online differentiation it 
offered. Ten programs offered multiple differentiated learning options. Nine programs provided teacher- 
directed differentiation of the online content, wherein teachers selected from multiple available programs 
or else controlled the content and difficulty level within a program to match a student’s learning needs 
(figure 2). Nine programs used adaptive software for differentiation, which monitors a student’s progress 
within the program and aligns the content, pacing, or difficulty of the material to a student’s individual 
needs. Six programs used both teacher-directed and computer adaptive differentiation. In 10 programs stu- 
dents were able to progress at a flexible pace so that they could take the time they needed to master the 
online content. 

All blended learning programs offered opportunities for the online component to inform the in-class 
curriculum. All 11 blended learning programs with studies that met the current study’s eligibility criteria 
provided progress reports or skills assessments that classroom teachers could use to inform their instruction. 


Figure 2. Of the 14 online and blended learning programs with studies that met the current 
study’s eligibility criteria, 9 provided teacher-directed differentiation of the online content, and 9 
used adaptive software for differentiation 

Number of programs 



Teacher directed Computer adaptive Student self-paced 

Note: Of the 14 programs, 10 offered more than one type of differentiation. 

Source: Authors' analysis in November 2015 following the procedures described in appendix A. 
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In eight blended learning programs the online component was intentionally sequenced with in-class instruc- 
tion. Five blended learning programs used the online component to offer in-class differentiation: teachers 
either used the time students spent on the computers to provide individual or small-group support to stu- 
dents not online or used students’ online performance to differentiate the in-class portion of the curriculum. 

Characteristics of studies of online and blended learning programs 

Of the 17 studies that examined online and blended learning programs and met the current study’s eligibil- 
ity criteria, 15 used a group comparison design (that is, a randomized controlled trial or quasi-experimental 
design; figure 3) and 2 were correlation studies. Among the 17 studies, the following student achievement 
outcomes measures were examined: 

• Academic motivation. 

• Attendance. 

• Course completion. 

• Course grade. 

• Course-specific end-of-course assessments. 

• CTB/McGraw-Hill Acuity proficiency exam. 

• Disciplinary actions. 

• Dynamic Indicators of Basic Early Literacy Skills. 

• Educational Testing Service end-of course assessments. 

• Math attitudes and confidence. 

• Stanford Achievement Tests. 

• State achievement tests. 

• Terra Nova assessments. 

• Test of Early Reading Ability-2. 

The 17 studies were classified according to whether the outcomes reported by their authors indicated sta- 
tistically significant positive, negative, mixed, or nonsignificant effects. Because most studies examined 


Figure 3. Of the 17 studies that examined online and blended learning programs and met the 
current study’s eligibility criteria, 15 used a randomized controlled trial or quasi-experimental 
design 

Number of studies 



Randomized controlled trial Quasi-experimental design Correlational method 


Source: Authors' analysis in November 2015 following the procedures described in appendix A. 
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multiple outcomes, a study was classified as having mixed effects only if it reported statistically significant 
positive and negative effects across or within outcome domains. If a study had both nonsignificant and 
significant effects within an outcome domain, it was classified according to the direction of the significant 
effect (positive or negative). Thirteen studies reported a statistically significant positive effect, and one 
study reported a negative effect from the use of online or blended learning programs that offered differen- 
tiated learning options. Two studies reported mixed effects, and one study did not report any significant 
effects. 

The quality of the 15 studies that met the current study’s eligibility criteria and used a randomized con- 
trolled trial or quasi-experimental design varied. Interpretations regarding the causal effects of the pro- 
grams examined should thus be made with caution. 

Seven group comparison design studies were rated by the study team as meeting WWC standards with 
or without reservations. Fifteen group comparison design studies (those that used a randomized controlled 
trial or quasi-experimental design) either had been previously reviewed in WWC intervention reports 
or were rated by the study team using decision rules modeled after WWC criteria. For studies that were 
reviewed in WWC intervention reports that met the current study’s eligibility criteria, this study used the 
rating reported in the intervention report. (There were other online and blended learning programs that 
were the subject of WWC intervention reports but that did not meet the current study’s eligibility criteria.) 

Of the 15 group design studies that met the current study’s eligibility criteria, six met WWC standards 
without reservations, and one met WWC standards with reservations. Of these seven studies, four had 
been the focus of WWC intervention reports, and three were rated by WWC-certified members of the 
study team. 

Two studies examined the effects of the Cognitive Tutor Algebra I blended learning program on students’ 
Educational Testing Service Algebra I end-of-course assessment and final course grades (Morgan & Ritter, 
2002; Ritter, Kulikowich, Lei, McGuire, & Morgan, 2007). Both studies were the focus of a WWC inter- 
vention report and were rated as meeting WWC standards without reservations. Both studies reported 
significant positive effect sizes related to assessment scores (0.29 and 0.38) and course grades (0.26 and 
0.36). Effect sizes describe the size of a treatment effect — in this case the difference between the scores of 
students who participated in online or blended learning programs and the scores of those who did not. See 
box 2 for details on how to interpret effect sizes. 

One study that the current study team determined met WWC standards without reservations assessed the 
impact of the Cognitive Tutor Geometry blended learning program (Pane, McCaffrey, Slaughter, Steele, & 
Ikemoto, 2010). The study reported a statistically significant negative effect size of -0.19 on students’ end- 
of-course assessment scores and nonsignificant effects on students’ math confidence and attitudes. 

Two studies, which were the focus of a WWC intervention report and were rated as meeting WWC 
standards without reservations, examined the effects of multiple blended learning programs on Stanford 
Achievement Test reading and algebra assessments. The second study (Campuzano, Dynarski, Agodini, & 
Rail, 2009) was conducted as a follow-up to the first (Dynarski et al., 2007). The studies assessed the follow- 
ing blended learning programs: Cognitive Tutor Algebra I, Destination Reading, LeapTrack, PLATO Focus, 
Read 180, and Waterford Early Reading. Dynarski et al. (2007) found no statistically significant effects. 
Campuzano et al. (2009) found a statistically significant positive effect of 0.09 of the LeapTrack program on 
students’ Stanford Achievement Test reading scores, but no other statistically significant effects. 
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Box 2. Effect sizes 


Effect size characterizes an effect against a common point of reference to allow for comparisons. The current 
study’s effect sizes are characterized using the standard deviation of the outcome (Dynarski & Kisker, 2014). 
The standard deviation can be interpreted as the average distance in either direction between students’ scores 
and the average score. A small standard deviation means that students are tightly clustered around the average 
score, and a large standard deviation means that students are spread more widely around the average score. 

A useful way to understand effect sizes for an intervention is to compare them to effect sizes for other more 
commonly understood differences. One such difference is the amount of change one might expect to see in a 
year of schooling. In one year of schooling for students in grade 4, an effect size for the amount of growth is 
estimated at 0.36 in reading and 0.52 in math (Hill, Bloom, Black, & Lipsey, 2008 [add to reference list]). If an 
effect size for an online or blended learning intervention in reading is 0.30, it can be interpreted as meaningful 
because the gain associated with participating in the intervention is nearly as large as the gain one might 
expect, on average, from a year of schooling. 

Statistical significance is a common way to judge the noteworthiness of the results of a research study. 
Statistical significance is influenced by both the effect size and the sample size. When the sample size is large, 
a small effect size will be significant. When the sample size is smaller, the effect size has to be larger to reach 
statistical significance. There can thus be some cases where a statistically nonsignificant finding has an effect 
size as large as 0.25 or -0.25. 


The study team determined that one study on the READ 180 blended learning program met WWC stan- 
dards without reservations (Kim, Capotosto, Hartry, & Fitzgerald, 2011). The study found a statistically 
significant positive effect on students’ Stanford Achievement Test assessment scores with an effect size of 
0.32 for reading comprehension and 0.23 for vocabulary. The study found no significant effects on students’ 
Stanford Achievement Test spelling or Dynamic Indicators of Basic Early Literacy Skills (fluency) scores. 

A study on the Time to Know blended learning program (Rosen & Beck-Hill, 2012) was rated by the study 
team as meeting WWC group design standards with reservations, only for the grade 5 reading outcome. 
Results of the study showed a statistically significant positive effect on students’ Texas Assessment of 
Knowledge and Skills reading scores, with an effect size of 0.23. Outcomes for math achievement, student 
attendance, disciplinary actions, and motivation to learn math and reading were rated as not meeting 
WWC group design standards. 

Two correlational studies examined the influence of student-teacher communication. The two cor- 
relational studies assessed the association of student-teacher interaction with student achievement out- 
comes in fully online, self-paced, virtual schools. The first study found a significant positive correlation 
between the frequency and quality of student-teacher interactions and course completion and course grade 
(Hawkins, Graham, Sudweeks, & Barbour, 2013). In that study, the quality of student-teacher interaction 
was operationalized as the degree to which student-teacher interaction consisted of academic feedback, 
course logistics information, and interpersonal support. The second study found a significant positive asso- 
ciation between the number of teacher comments provided and an Algebra I end-of-course assessment and 
significant negative association between the number of teacher comments provided and an Algebra II end- 
of-course assessment (Liu & Cavanaugh, 2012). 
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Implications of the study findings 


Of the 162 studies identified that examined the association of online and blended learning programs that 
offer differentiated learning options with student academic outcomes, only 17 met the current study’s eli- 
gibility criteria. The remaining studies did not meet eligibility criteria because they did not focus on K-12 
programs, assessed standalone online programs, or did not clearly offer differentiated learning options. Of 
the 17 studies only 7 used a rigorous group research design to assess the impact of online and blended 
learning programs, making it difficult to draw conclusions about the effectiveness of online and blended 
learning programs. 

Even though the findings in this report do not allow for conclusions to be drawn about the effectiveness of 
these programs as a whole, educators may use the findings to consider individual online or blended learning 
programs for which there is evidence of influencing student achievement. The findings might also inform 
educators’ decisionmaking about online and blended learning programs. For example, this report can help 
educators consider the form of online or face-to-face differentiation that they want an online or blended 
learning program to provide, as well as the role that the face-to-face teacher will play. 

Limitations of the study 


The current study is limited to findings from studies identified using the keyword search process and screen- 
ing criteria outlined in appendix A. This report summarizes only programs with studies whose report nar- 
ratives clearly indicated that the programs were used to offer differentiated learning options. It is possible 
that programs that were not included in the current study could be used to support differentiated instruc- 
tion, but the screening criteria used for the current study did not identify studies about those programs. 
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Appendix A. Literature search procedures 


To address the research questions, the study team conducted a comprehensive search to identify relevant 
articles. The search proceeded in two steps: an electronic database keyword search and a program search. 
After completing these two steps, the study team consulted with experts in the field, such as staff from the 
International Association for K-12 Online Learning, to identify any other articles that should be included. 

Electronic database keyword search 

The first step in identifying research on online and blended learning programs was the electronic database 
keyword search. Three members of the study team used keywords to search three electronic databases, Aca- 
demic Search Premier, ERIC, and PsycINFO, in February-May 2015. The members of the study team were 
trained to screen studies and keep those that met the following relevance criteria: 

• Must be a study or a literature review of studies associated with student academic outcomes. 

• Must use a quantitative research design, such as a randomized controlled trial, quasi-experimental 
design, or correlational method. 

• Must examine an online or blended learning program (standalone programs were excluded). 

• Must examine a program that allows students to proceed at their own pace through course content, 
individualizes course content or difficulty level, or supports face-to-face classroom differentiation (a 
program was included if it had at least one of these characteristics). 

• Must involve a K-12 public school in the United States, including traditional schools, charter 
schools, and magnet schools. 

• Must be published in 1995 or later. 

• Must be written in English. 

• Must be published in an academic journal or publicly available (dissertations and theses were 
excluded). 

Search terms included 30 keywords related to online and blended learning, 5 keywords related to differen- 
tiated learning options or personalized instruction, and 9 keywords related to quantitative research. The 
initial keywords were developed in consultation with members of Regional Educational Laboratory (REL) 
Central’s Technical Working Group, a standing group charged with providing advice or guidance on REL 
Central’s work. As the keyword search progressed, additional keywords that were identified were added to 
the initial list (the final list of keywords is presented in box Al). Keyword searches were conducted using 
every combination of online/blended learning, differentiation, and quantitative research keyword. 

The study team conducted an initial keyword search and screened the study titles and abstracts to deter- 
mine whether the studies met the relevance criteria. If it was unclear from the information contained in 
the title and abstract whether a study met the relevance criteria, the study team included the study to be 
more closely reviewed in the second stage of screening. As a result of the electronic database keyword 
search and initial screening, 124 potentially relevant studies were identified (figure Al). Thirty-eight addi- 
tional studies were identified through a subsequent gray literature search (see below), resulting in the review 
of a total of 162 studies. 

The study team conducted a second screening of the potentially relevant reports. Before the screening, the 
study team screened a common set of 10 studies and compared their results. When decisions differed, the 
study team discussed the rationale for their decisions. An additional set of five studies were screened; for 
these studies the study team had no disagreement in the decisions. 
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Box Al. Final keyword search terms 


Online and blended learning terms 

• “online learn*” 

• 

“electronic class*” 

• 

“online instruct*” 

• “blended class*” 

• 

“electronic instruct*” 

• 

“online school*” 

• “blended instruct*” 

• 

“electronic learn*” 

• 

"virtual class*" 

• “blended learn*” 

• 

“electronic school*” 

• 

"virtual instruct*" 

• “blended school*" 

• 

“hybrid class*” 

• 

"virtual learn*” 

• “cyber class*” 

• 

“hybrid instruct*” 

• 

“virtual school*” 

• “cyber instruct*” 

• 

“hybrid learn*” 

• 

“web-based class*” 

• “cyber learn*” 

• 

“hybrid school*” 

• 

“web-based instruct* 

• “cyber school*” 

• 

“intelligent tutoring” 

• 

“web-based learn*” 

• “e-learning” 

• 

“online class*’’ 

• 

“web-based school*” 

Differentiation terms 

• differentiate* 

• individualize* 

• 

• 

personalize* 

“self-pace*” 

• 

“self-guide*” 

Research terms 

• “comparison group” 

• 

baseline 

• 

random 

• “control group” 

• 

correlation 

• 

study 

• “quasi-experiment*” 

• 

experiment 

• 

treatment 


Note: Quotation marks indicate that the study team searched for the exact phrase (for example, "intelligent tutoring” was 
searched as one term in which both words in sequence needed to be present). Asterisks indicate wildcards, used so that 
words with different endings could be found (for example, “learn*” would produce results that include the words “learner” and 
“learning”). 

Source: Authors' compilation. 


Figure Al. Number of studies considered at each stage of the screening process 


Initial title and abstract screen: 124 studies 


Second screening: 11 studies 


Gray literature search identified 38 additional studies, 6 met eligibility criteria 


Final screen: 17 studies 


Source: Authors' analysis in November 2015 following the procedures described in appendix A. 


During the second screening stage, the study team consulted the full text of each document as necessary to 
determine whether it was indeed a report of a research study (defined as a study that presented quantitative 
data collected using clearly defined methods), whether the study was conducted with a sample of K-12 
students in the United States, and whether the study addressed an online or blended learning program. 
Studies were screened out if the program was a standalone program or if there was no evidence that the 
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teachers integrated the program into their classroom curriculum. Studies were also screened out if the 
program was not designed to provide differentiated learning options or if there was no evidence that the 
program was used for this purpose in the study. When the screening decision was uncertain, the study team 
discussed the ways in which each study met or did not meet each criterion until they reached consensus 
about the final screening decision. During this round literature reviews and meta-analyses were screened 
out, but studies found within them that had not previously been identified were screened. Additionally, 
the names of online and blended learning programs examined by the screened studies or referred to in the 
study narrative were recorded. After this second screening, 11 articles were determined to meet all study 
criteria. 

Program search 

On the basis of the review of studies in the second stage of screening and after consultation with Technical 
Working Group members, the study team developed a list of 37 online and blended learning programs (box 
A2). The study team searched the same three databases that were used for the keyword search for these 
names and also searched the websites of their vendors for additional possible studies. This process identified 
38 additional documents, 6 of which met all screening criteria. 

Final screening 

During the final screening, the study team carefully read each study in its entirety, summarized it, and 
identified the program type and research method categories. During this process the documents were again 
screened to ensure that the studies examined online or blended learning programs and not standalone pro- 
grams and that the programs offered differentiated learning options. This identified the 17 studies examin- 
ing 14 blended and online learning programs described in this report. 


Box A2. Names of online and blended learning programs 


Academy of Reading 

Achieve Now 

Achieve3000 

Algebra I Online 

Alliance College-Ready Public 

Schools 

Apangea/ThinkThroughMath 
Apex Learning 
Assessment to Instruction 
BrainHoney 
Cognitive Tutor 1 
Compass-Odyssey 
Destination Learning (was 
Destination Reading) 


Dream Box 
EdTech 
Firstline 
Headsprout 

Hybrid Learning Institute 

iLearn Math 

Istation 

Khan Academy 

KIPP Empower Academy 

Knewton 

Knowledgebox 

Larson Pre-Algebra I 

LeapTrack 

New Tech Network 


Plato Focus 

Read 180 

Reading Plus 

Revolution K12 

RocketShip 

ST Math 

SuccessMaker 

TenMarks 

ThinkPort 

Time To Know 

Waterford Early Reading 

Program 


Note 

1. Includes Cognitive Tutor Algebra I and Cognitive Tutor Geometry. 
Source: Authors' compilation. 
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Appendix B. Online and blended learning programs and 
research studies included in the literature summary 


This appendix provides a summary of the 14 online and blended learning programs and the 17 studies 
examined in this report (tables B1 and B2). Each program is described in terms of its online and in-class 
characteristics. Each study is described in terms of the research design used, independent and dependent 
variables, and study findings. 
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Table Bl. Characteristics of the online and blended learning programs examined in studies that met the current study’s eligibility criteria 



Online program characteristics 




In-class program characteristics 


Citation 

Program 

name 

Teacher 

Online individualizes 

communication content 

Program 

individualizes 

content 

Students 

self-pace 

Online 

assessment 

Program 
data may 
inform 
instruction 

Online 
integrated 
into class 
curriculum 

Differentiation 
or skill 
grouping 

Grade or 
topic 

Content area 

Full curriculum programs that use a blended learning approach 

Campuzano 
et al. (2009) 

PLATO Focus 

None ✓ 



Progress report 


✓ 


Elementary 

English 
language arts 

Campuzano 
et al. (2009); 
Dynarski et al. 
(2007); Morgan 
& Ritter (2002); 

Pane et al. 

(2014); Ritter 
et al. (2007) 

Cognitive 

Tutor Algebra 1 

None 

✓ 

✓ 

Progress report 

Skill assessment 

✓ 

✓ 


Algebra 

Math 

Murphy, 

Mislevy, 
Gallagher, 
Krumm, & Wei 
(2014) 

Alliance 

College- 
Ready Public 

Schools 

None ✓ 

✓ 

✓ 

Progress report 

✓ 

✓ 

✓ 

High 

school 

Math and 

English 
language arts 

Murphy et al. 
(2014) 

Knowledge 
is Power 

Program 

Empower 

Academy 

None 

✓ 

✓ 

Progress report 



✓ 

Elementary 

Math and 
English 
language arts 

Pane et al. 

(2010) 

Cognitive 

Tutor 

Geometry 

None ✓ 

✓ 


Progress report 

Skill assessment 

✓ 

✓ 


Geometry 

Math 

Rosen & 

Beck-Hill 

(2012); Weiss & 
Rosen (2010) 

Time To Know 

None ✓ 

✓ 


Ongoing 

formative & 

summative 

✓ 

✓ 

✓ 

Elementary 

Math and 

English 
language arts 

Supplemental curriculum programs that use a blended learning approach 

Campuzano 
et al. (2009); 
Dynarski et al. 
(2007) 

Destination 

Reading 

None ✓ 



Progress report 

✓ 



Elementary 

English 
language arts 

(continued) 





Table Bl. Characteristics of the online and blended learning programs examined in studies that met the current study’s eligibility criteria 

(continued) 





Online program characteristics 




In-class program characteristics 


Citation 

Program 

name 

Teacher 

Online individualizes 

communication content 

Program 

individualizes 

content 

Students 
self pace 

Online 

assessment 

Program 
data may 
inform 
instruction 

Online 
integrated 
into class 
curriculum 

Differentiation 
or skill 
grouping 

Grade or 
topic 

Content area 

Campuzano 
etal. (2009); 
Dynarski et al. 
(2007) 

LeapTrack 

None 

✓ 

✓ 

✓ 

Skill assessment 

✓ 

✓ 


Elementary 

English 
language arts 

Campuzano 
etal. (2009); 
Dynarski 
et al. (2007); 
Cassady & 

Smith (2003, 
2005); Powers 

& Price-Johnson 
(2006); Tracey 
& Young (2007) 

Waterford 

Early Reading 

None 

✓ 

✓ 

✓ 

Progress report 

Skill assessment 

✓ 

✓ 


Elementary 

English 
language arts 

Dynarski et al. 
(2007); Kim 
etal. (2011) 

Read 180 

None 


✓ 

✓ 



✓ 

✓ 

Elementary 

English 
language arts 

Murphy et al. 
(2014) 

Firstline 

Schools 

None 

✓ 

✓ 

✓ 

Progress report 

✓ 


✓ 

Elementary 

Math and 

English 
language arts 

Full curriculum programs that use an online learning approach 

Plawkins et al. 

(2013) 

Utah 

Electronic 

H igh School 

Asynchronous 



✓ 

Graded 

assignments 

na 

na 

na 

High 

school 

All 

Liu & 

Cavanaugh 

(2012) 

Virtual School 

Asynchronous 



✓ 

Course 

assessments 

na 

na 

na 

Algebra 

Math 

Shoaf (2007) 

Calvert 

School 

Asynchronous 

✓ 


✓ 

Course 

assessments 

na 

na 

na 

Elementary 

Math and 

English 
language arts 


na is not applicable because there is no in-class component in the online program. 

Note: Asynchronous online communication refers to communications that are not conducted in real time (for example, communication that is conducted only over email). 
Source: Authors' analysis in November 2015 following the procedures described in appendix A. 





Table B2. Characteristics of the 17 studies that met the eligibility criteria of the current study 







What Works 

Citation 

Program 

name 

Program 

type 

Independent 

variable 

Outcome measure 

Findings or conclusions 

Clearinghouse 
(WWC) rating, 
if applicable 

Randomized controlled trial 

Morgan 

Cognitive 

Blended 

Group 

Educational Testing 

Significant positive 

Meets WWC 

& Ritter 

Tutor Algebra 1 


assignment 

Service Algebra 1 end-of- 

effects on both outcomes 

standards 

(2002) 




course assessment 


without 





Course grade 


reservation 

Pane et al. 

Cognitive 

Blended 

Group 

End-of-course 

Significant negative 

Meets WWC 

(2010) 

Tutor 


assignment 

assessment. Student 

effect on end-of-course 

standards 


Geometry 



math confidence and 

assessment 

without 





attitudes 

Nonsignificant effects 

on math confidence and 

attitudes 

reservations 

Pane et al. 

Cognitive 

Blended 

Group 

CTB/McGraw-Hill Acuity - 

Significant positive effect 

Does not meet 

(2014) 

Tutor Algebra 1 


assignment 

Algebra Proficiency Exam 

on high school math 

WWC standards 





Student math confidence 

proficiency 






and attitudes 

Nonsignificant effects 

on math confidence and 

attitudes and middle 

school math achievement 


Ritter et al. 

Cognitive 

Blended 

Group 

Educational Testing 

Significant positive effect 

Meets WWC 

(2007) 

Tutor Algebra 1 


assignment 

Service Algebra end-of- 

on course grade 

standards 





course exam 

Course grade 

Nonsignificant effect on 

course exam 

without 

reservations 

Campuzano 

Multiple 

Blended 

Group 

Stanford Achievement 

No significant effects for 

Meets WWC 

et al. 

Interventions 


assignment 

Test - Reading 

four interventions 

standards 

(2009) 




Education Testing Service 
- Algebra 

Significant positive effect 

for one intervention 
(LeapTrack) 

without 

reservations 

Dynarski 

Multiple 

Blended 

Group 

Stanford Achievement 

No significant effect 

Meets WWC 

et al. 

interventions 


assignment 

Test - Reading 


standards 

(2007) 




Education Testing Service 
- Algebra 


without 

reservations 

Kim et al. 

READ 180 

Blended 

Group 

Dynamic Indicators of 

Significant positive 

Meets WWC 

(2011) 



assignment 

Basic Early Literacy Skills 

effects on reading 

standards 





- Oral Reading Fluency 

comprehension and 

without 





Stanford Achievement 

vocabulary 

reservation 





Test - Reading 

No significant effects 






comprehension, 

on reading fluency and 






vocabulary, spelling 

spelling 


Quasi-experimental design 

Shoaf 

Calvert 

Online 

Group 

Ohio State Achievement 

No test of statistical 

Does not meet 

(2007) 

School 


assignment 

Test - math and English 

significance available 

WWC standards 





language arts 

Positive effects 

compared to comparison 
school, negative effects 
compared to state 
average 


(continued) 
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Table B2. Characteristics of the 17 studies that met the eligibility criteria of the current study (continued) 

Citation 

Program 

name 

Program 

type 

Independent 

variable 

Outcome measure 

Findings or conclusions 

What Works 
Clearinghouse 
(WWC) rating, 
if applicable 

Murphy 

Multiple 

Blended 

Group 

California Standards Test 

No significant effect for 

Does not meet 

et al. 

(2014) 

interventions 


assignment 

Terra Nova - math and 

reading 

Northwest Evaluation 

Association's 

Measurement of Academic 

Progress- English 
language arts and math 

three interventions 

Significant positive effects 

for two interventions 

in math and English 
language arts (Firstline 
schools and Knowledge is 
Power Program Empower 
Academy) 

WWC standards 

Rosen & 

Beck-Hill 

(2012) 

Time To Know 

Blended 

Group 

assignment 

Texas Assessment of 

Knowledge and Skills - 
math and reading 

Student attendance, 
disciplinary actions, 

motivation to learn math 

and reading 

Teacher use of 

differentiated instruction 

Significant positive 

effects on all outcomes 

Meets WWC 

standards with 

reservations for 

grade 5 reading. 
All other 

outcomes do 

not meet WWC 

standards. 

Weiss & 

Rosen 

(2010) 

Time To Know 

Blended 

Group 

assignment 

Texas Assessment of 

Knowledge Skills - math 

Math reasoning 

assessment 

Significant positive 

effects on all outcomes 

Does not meet 

WWC standards 

Cassady 

Waterford 

Blended 

Group 

Phonological Awareness 

Significant positive 

Does not meet 

& Smith 

(2003) 

Early Reading 


assignment 

Test 

Concepts About Print 

effects on both outcomes 

WWC standards 

Cassady 

& Smith 

(2005) 

Waterford 

Early Reading 

Blended 

Group 

assignment 

Terra Nova 

Comprehensive Test of 

Basic Skills 

Significant positive effect 

Does not meet 

WWC standards 

Powers 

Waterford 

Blended 

Group 

Dynamic Indicators of 

Significant positive 

Does not meet 

& Price- 

Johnson 

(2006) 

Early Reading 


assignment 

Basic Early Literacy Skills 
- initial sound fluency, 
letter naming fluency, word 
use fluency, phoneme 
segmentation fluency, 
nonsense word fluency 

District Core Curriculum 

Standard Assessment 

reading 

effects on all outcomes 

WWC standards 

Tracey 

Waterford 

Blended 

Group 

Test of Early Reading 

Significant positive effects 

Does not meet 

& Young 
(2007) 

Early Reading 


assignment 

Ability 2 

Lindamood Auditory 
Conceptualization 

Waterford Reading 
Inventory 

on Test of Early Reading 
Ability 2 and Waterford 
Reading Inventory 

No significant effect 
on Lindamood Auditory 
Conceptualization 

WWC standards 
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Table B2. Characteristics of the 17 studies that met the eligibility criteria of the current study (continued) 


Citation 

Program 

name 

Program 

type 

Independent 

variable 

Outcome measure 

Findings or conclusions 

What Works 
Clearinghouse 
(WWC) rating, 
if applicable 

Correlational 

Hawkins 

et al. 

(2013) 

Utah 

Electronic 

High School 

Online 

Student- 

teacher 

interaction 

frequency & 
quality 

Course grade and 
course completion 

Significant positive effect 

na 

Liu & 

Cavanaugh 

(2012) 

Virtual School 

Online 

Number 

of teacher 

comments 

End-of-course 

assessment 

Significant positive 
effect in Algebra 1 

Significant negative 
effect in Algebra II 

na 


na is not applicable. 

Source: Authors' analysis in November 2015 following the procedures described in appendix A. 
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Notes 


1. Only WWC'Certified reviewers working on a WWC contract are able to complete WWC reviews that 
are recorded in the WWC reviewed studies database. For this study, if official WWC ratings were 
not available, certified reviewers applied the WWC standards to determine potential study ratings. 
Determinations made by researchers for this study are not official WWC ratings until the studies are 
reviewed by WWC contractors and recorded in the WWC database of reviewed studies. 

2. WWC intervention reports contain reviews of multiple studies. Many of these studies were not 
addressed in the current study. WWC group design ratings were selected for only those studies examin- 
ing online and blended learning programs that met the eligibility criteria of the current study. 
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